The present invention relates to scoring of detection systems, in general, and to error-derived scoring of detection systems, in particular.
A detection system is a system that receives an input signal and has to detect a property of the signal, typically whether the source of the signal is a predefined object. Such systems may include radar systems, speaker recognition systems, medical diagnostic systems, fingerprint identification systems and many more. In speaker recognition systems, for example, the input signal is a collection of speech samples, and the property to be detected is the identity of the speaker. See, for example, A. Rosenberg and F. K. Soong, xe2x80x9cRecent Research in Automatic Speaker Recognitionxe2x80x9d, Advances in Speech Signal Processing, S. Furui and M. M. Sondhi, Eds., Marcel Dekker Inc., 1991 and U.S. Pat. No. 5,926,555 to Ort et al.
Reference is made to FIG. 1, which is a block diagram illustration of a detection system, generally referenced 100, as is known in the art. Detection system 100 comprises a detection unit (DU) 102 and an application system (AS) 104. AS 104 issues a detection command to DU 102 with the necessary parameters. AS 104 must supply DU 102 with one or more claimed identities of the target. DU 102 collects the input signal, transforms it to some internal representation and compares the internal representation with the claimed identities. The detection result is sent to AS 104, which performs a predefined action according to this result. In speaker recognition systems, for example, the action can be to grant the speaker access to some service, or to reject the speaker as an impostor.
In many detection systems, the detection result of DU 102 is not a binary positive/negative decision, but rather a numerical score. This score indicates the certainty of the suggested decision. A commonly used score is the posterior probability that defines the probability of detecting the searched property given the input signal, as is described in R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis, Wiley, 1973, pp. 10-13. In speaker recognition systems, for example, this score can be the probability that the given speech samples were generated by the claimed identity speaker.
When AS 104 has to choose an action based on the detection result, the numerical score is translated into a binary positive/negative decision. This is done by setting a threshold, so that the decision made (and the action performed) depend upon whether the score is above or below the threshold.
The threshold is set after considering many factors, such as the prior probabilities of having each result (the probabilities are known to AS 104 in advance), and the cost of the different possible errors. Two general types of errors are possible. A xe2x80x9cfalse positivexe2x80x9d error happens when a positive decision is made by AS 104 while the correct decision (by some criterion other than a comparison with the threshold) is negative. A xe2x80x9cfalse negativexe2x80x9d error happens when a negative decision is made by AS 104 while the correct decision (by some criterion other than a comparison with the threshold) is positive.
For example, an application subsystem of a speaker recognition system in a bank may decide to grant access to some data only to speakers who scored more than 0.99. The threshold is set so high because it is known that the cost of showing the data to an impostor is very high, but the cost of rejecting a true speaker is not that high because he can try again. To demonstrate the influence of prior probability on the setting of the threshold, this threshold may be increased at nighttime to 0.995 reflecting the prior knowledge that most breaking in events happen at night.
Reference is now made to FIG. 2, which is an illustration of a graph showing the count of scores as a function of the score value, for a hypothetical detection system, as is known in the art. A threshold 200 divides the scores into positive scores 201 (scores higher than the threshold 200 lead to a positive decision by an application system) and negative scores 202 (scores lower than the threshold 200 lead to a negative decision by the application system). A distribution 203 of scores for which the correct decision is positive, is indicated with a dashed-line curve. Similarly, a distribution 204 of scores for which the correct decision is negative, is indicated with a solid-line curve.
A hatched area 205 indicates the rate of false negative errors, i.e. scores for which the correct decision is positive and which lead to a negative decision by the application system. A hatched area 206 indicates the rate of false positive errors, i.e. scores for which the correct decision is negative and which lead to a positive decision by the application system.
Choosing the threshold 200 inaccurately may result in sub-optimal use of the detection system. If the threshold is too low, there may be too many false positive errors. The reduction of false negative errors in this case does not necessarily compensate for the increase of the false positive errors. Rather, the relative significance of false positive errors and false negative errors is expressed by the cost function. Similarly, if the threshold is too high, there may be too many false negative errors.
The calculation of the optimal threshold 200 can be very difficult, especially when the internals of the detection system are hidden from the user. On the other hand, the detection system cannot output a binary positive/negative decision when the priors of the input and the cost function are not known to the designer of the detection system.
There is provided in accordance with a preferred embodiment of the present invention a method for use in a detection unit that produces a score to be converted into a binary decision via the setting of a threshold. The method is a method for generating the score as an error-derived score such that the threshold is a tolerable one-sided error probability. The method includes the steps of generating a primary score that is a monotonic function of the posterior probability, obtaining a distribution of primary scores of input signals that ought to lead to a particular binary decision, and translating, based on the distribution, the primary score of a current input signal to the error-derived score.
In accordance with a preferred embodiment of the present invention, either the distribution is determined from knowledge of internals of the detection unit or from calibration input signals.
Moreover, in accordance with a preferred embodiment of the present invention, when the tolerable one-sided probability is a false positive error probability and the particular binary decision is a negative decision, the step of translating includes the step of determining the percentage of the input signals whose primary scores are higher than the primary score of the current input signal. Furthermore, in accordance with a preferred embodiment of the present invention, when the tolerable one-sided probability is a false negative error probability and the particular binary decision is a positive decision, the step of translating includes the step of determining the percentage of the input signals whose primary scores are lower than the primary score of the current input signal.